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Abstract: In this paper, we discuss a scalable approach for integrating learning analytics into an 
online K-12 science curriculum. A description of the curriculum and the underlying pedagogical 
framework is followed by a discussion of the challenges to be tackled as part of this integration. 

We include examples of data visualization based on teacher usage data along with a 
methodology for examining an inquiry-based science program. With more than one million 
students and fifty thousand teachers using the curriculum, a massive and rich dataset is 
continuously updated. This repository depicts teacher and student usage, and offers exciting 
opportunities to leverage data to improve both teaching and learning. In this paper, we use data 
from a medium-sized school district, comprising 53 schools, 1,026 teachers, and nearly one-third 
of a million curriculum visits during the 2012-2013 school year. This growing dataset also poses 
technical challenges such as data storage, complex aggregation, and analyses with broader 
implications for pedagogy, big data, and learning. 
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1. INTRODUCTION 

Computers, tablets, and mobile devices offer new tools to support learning, both inside and outside the 
classroom. This development, in turn, has created new opportunities to understand and assess their 
impact on educators and learners through the collection and analysis of user interaction data. Indeed, a 
major advantage of online curricula over traditional textbook-based curricula is the ability to capture 
these user interaction or learning analytics data. Learning analytics is a relatively new, but rapidly 
growing area of research (First International Conference on Learning Analytics and Knowledge, 2011), 
whose goal, according to the Society of Learning Analytics, is "understanding and optimizing learning 
and the environments in which it occurs." Most learning analytics (LA) methods use data generated by 
the interactions between learners and the Learning Management Systems (LMSs) they use (Pahl, 2004). 

Ongoing research in educational data mining and LA suggests that harnessing analytics data can advance 
new research methodologies in education, help educators better assess their pedagogical practices, and 
devise innovative educational methods, all with the goal of improving education (Elias, 2011; Campbell 
& Oblinger, 2007; Pardo & Delgado, 2011; Siemens, 2012). LA can leverage improvement through 
improved educational decision-making (Arnold, 2010; Norris, Baer, Leonard, Pugliese, & Lefrere, 2008; 
Vatrapu, Tplovs, Fujita, & Bull, 2011), clearer institutional and individual goal setting (Flendricks, Plantz, 
& Pritchard, 2008), more timely and frequent feedback for students and teachers (Fla, Bae, & Park, 
2000; Flamalainen, Suhonen, Sutinen, & Toivonen, 2004; Merceron & Yacef, 2005; Suthers, Ravi, 
Medina, Joseph, & Dwyer, 2008), the individualization of teaching and learning (Beck & Mostow, 2008; 
Farzan, 2004; Fleraud, France, & Mille, 2004; Lu, 2004; Talavera & Gaudioso, 2004), and the generation 
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of a richer set of data on student behaviour and learning (Baker & Yacef, 2009; Mazza & Dimitrova, 
2004; Pavlik, Cen, & Koedinger, 2009). 

With the rapidly growing interest in and technical ability to leverage these data in educational settings, 
there also is a growing sense that many recent educational technology and big data initiatives are 
detached from what we know about teaching and learning, particularly in the K-12 sector (Junco, 2012; 
Romero & Ventura, 2010). While the use of LA has been particularly fruitful in higher education and in 
online courses where many to all learning activities and interactions take place online, in conventional 
brick and mortar schools, it remains crucial to incorporate information derived from the other activities 
that go on in a classroom in order to capture fully the variety of learning experiences taking place. This is 
particularly important in science education where most experts agree that students need to be "doing" 
science: conducting experiments, making observations, and building models (Crawford, 2000; Haury, 
1993; Keys & Bryan, 2001). Without data on hands-on activities and classroom discussions, it can be 
hard to make sense of the LA data gathered by an LMS. It is, therefore, imperative to ensure that the use 
of LA in the K-12 sector is grounded in relevant pedagogies and serves as a complement to data on face- 
to-face interactive learning experiences not captured by digital artifacts such as student inquiry, 
classroom discussions, and hands-on activities. This comprehensive, mixed methods approach ensures 
that data analyses in LA are not reduced to the mere analysis of clicks and page visits. 

Furthermore, LA must be leveraged in ways that both recognize and draw on existing K-12 education 
research. While LA has an established presence in the business sector and even in higher education, its 
foray into K-12 education is particularly new and is being driven in large part by the start-up community 
and venture capital, and less by educators or educational researchers (Culatta, 2012; DeSantis, 2012). 
Although there is nothing wrong with these drivers, educators are unlikely to buy into new technology 
and big data initiatives if they are not immediately accessible and relevant to their daily work (Means, 
Padilla, & Gallagher, 2010; Watters, 2012; Wayman, Cho, & Richards, 2010). 

This paper expands our ongoing research (Monroy, Snodgrass Rangel, & Whitaker, 2013) and addresses 
this gap in the literature by outlining and discussing one strategy for incorporating LA into a broader 
investigation of how a K-12 online science curriculum is implemented in classrooms. The purpose of our 
implementation study is to increase teacher and student use of the curriculum through improved 
design. The strategy is meant to answer the following three key questions about the curriculum: 

1. How can we incorporate the LA data into a broader effort to understand how a digital 
curriculum is used? 

2. How can we leverage the LA data to support teachers and administrators? 

3. How can we integrate the LA data into the evaluation of the curriculum's impact? 

We have developed a three-pronged strategy to address these guiding questions: a technical prong 
related to the architecture supporting the collection and management of LA data, a quantitative prong 
related to how the LA data can be utilized to understand the curriculum's use and improve its design, 
and a final qualitative prong where we collect and analyze qualitative data on teacher (and in the future, 
student) use of the curriculum. 

We begin our discussion by briefly describing the curriculum examined here and its underlying 
pedagogical model. The technical architecture is explained in section three. In section four, we describe 
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several specific techniques we have developed to use the LA data to measure use of the curriculum and 
provide examples using real data from a school district in Texas. Here, we also discuss several 
considerations related to the normalization of data. In section five, we describe ways to use data- 
visualization techniques to support teachers and administrators. Then, in section six, we situate our use 
of LA data within our broader data strategy for measuring the implementation of the curriculum. In 
order to demonstrate how this might be accomplished, we provide examples from real data collected in 
the fall of 2012 from a second district in Texas. Finally, we end with a discussion about the future of data 
analytics in K-12 education, focusing in particular on the challenges we need to overcome and the 
potential for educational improvement. 


2. BACKGROUND: STEMSCOPES AND THE 5E+I/A PEDAGOGICAL 
FRAMEWORK 


STEMscopes is a comprehensive online K-12 science curriculum aligned to Texas state standards and 
grounded in the 5E science inquiry model. The curriculum offers a range of hands-on inquiry activities, 
online activities, and additional supporting resources for teachers for each of the learning standards at 
each grade level. STEMscopes currently serves about 2,400 schools, more than 50,000 teachers, and 
over one million students in Texas. These figures represent approximately 40 percent of the state's 
school districts, making it the most used science curriculum in the state. During the 2012-2013 school 
year, we gathered more than one hundred million data points generated by districts, schools, and users, 
both teachers and students. We were able to identify the science standard covered, the grade level, 
specific steps in the 5E+I/A step, materials accessed, and the weekly assessment results. 

The 5E pedagogical model is an inquiry-based learning approach that aims to harness learners' prior 
knowledge while eliciting interest in new phenomena (Bybee et al., 2006). This instructional model was 
created and refined by the Biological Science Curriculum Study (BSCS). The 5E acronym stands for the 
following five steps that comprise a full lesson cycle: engage, explore, explain, elaborate, and evaluate. 
In the Engage phase, students draw on prior knowledge to raise their interest in and activate their prior 
knowledge of the new content. During the Explore phase, students take part in activities and 
experiments that allow them to experience and learn new concepts and skills through active 
investigations. The Explain phase requires students to explain those new concepts and skills in their own 
words. New experiences in the Elaborate phase deepen student understanding of the new concepts. 
Finally, learner understanding is assessed in the Evaluate phase. Teachers are meant to implement 
lessons following this cycle in order to optimize learning. 

The STEMscopes curriculum adds two steps to the basic 5E model: one for intervention and one for 
acceleration. Intervention activities are designed to support those students still struggling to master the 
standards, while the acceleration activities are designed for those students who have mastered the 
concepts and are ready to extend and apply their learning (Whitaker, 2012; Zuiker & Whitaker, 2013). 
With these two additional steps, the model becomes the 5E+I/A model. 
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Engage 


Demonstration Presentation Teacher Instructions 


This element provides the necessary information for teachers 
to conduct the Engage Demonstration Presentation activity. 
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A pre-made slide presentation to elicit prior knowledge and 
excite students about learning the topic. 
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and do not know before instruction on the topic. 
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Figure 1: Screen shot of a science unit titled, "Changes from Heat." The column on 
the left presents elements grouped by each 5E phase (Engage, Explore, Explain, 

Elaborate, and Evaluate). 
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In addition to the online activities and materials, the curriculum also offers accompanying kits that 
contain the science materials necessary to do every hands-on activity in the curriculum. The kits allow 
students to conduct science experiments, build models, create simulations, and make observations, 
thereby offering them a deeper learning experience meant to re-create the work of scientists (Haury, 
1993; Minner, Levy, & Century, 2010). In the Texas version of the curriculum 1 , there are nearly four 
hundred standards or "scopes" in total across all grade levels. Figure 1 provides an example of what the 
5E+I/A model looks like in the curriculum. Specifically, it depicts the 4 th grade standard, "Changes from 
Heat." Components or activities for each of the seven (5E+I/A) steps are presented in the column on the 
left. They include materials for teachers such as background information on the learning standard, 
interactive games, and assessments. 


component and “Description 


0 


Science Rock 

Students listen to and "dance" (using a computer keyboard or 
USB attached dance mat) to a song called "Changes and 
Properties." 


Demonstration Presentation 

Students will observe how water condenses on the outside of a 

<u 

oo can. 

re 

oo 

uj Pre-Assessment 

Students answer five multiple choice questions drawing on 
general prior knowledge. 


Starters 

Introductory activities designed to engage students to think 
about changes from heat. 


0) 

_2 Explore 

^ Students learn about insulation and condensation. 


Question Prompts 

Question ideas specifically related to the Explore activity. 

c Vocabulary 

2 Picture Vocabulary with matching Student Vocabulary Cards 
g- and an Interactive Vocabulary Game to learn and review key 
^ vocabulary for changes from heat. 

Progress-Monitoring Assessment 

A five-question multiple choice assessment designed to assess 
the student's mid-unit concept attainment. 


Interactive Vocabulary Game 

An Inte/detae game tte 


>n Be played win a elm or moWMVWV to 


. wt»rd|_J8® 

S&fARI\ 




■r in the bag WITH 


Figure 2. Various elements for one science standard: 1) a partial summary of the content of 
one unit, 2) an interactive game, and 3) a student journal. 


Figure 2 shows three elements within a scope: 1) an overview of a unit's content in the context of the 
5E+I/A model, which provides a snapshot of all the components a teacher can find for that scope and is 
intended for class planning purposes; 2) an interactive game in the Explain phase that offers an engaging 


1 There also is a national version created to help teachers implement the Next Generation Science Standards 
(NGSS), but the NGSS version does not follow the 5E+I/A model. 
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opportunity for students to practice new content vocabulary; and 3) a student journal sheet that can be 
used by students when completing hands-on activities as part of the Explore step. 


3. AN ARCHITECTURE FOR SUPPORTING LEARNING ANALYTICS 


As an online curriculum, STEMscopes creates an environment where data is constantly generated and 
stored as teachers and students use the curriculum on a daily basis. For this reason, the architecture 
enables the curation, storage, and processing of that large amount of data. As an example, during the 
2012-2013 school year, nearly a quarter of a million interactions with the curriculum were captured 
each day. This volume of data demands techniques from the areas of databases, data warehouses, and 
big data. We have designed a distributed data warehouse environment that allows us to scale the 
storage and processing of user-generated data. 

To serve our storage and computing needs, we employ four separate high-end computer servers from a 
cloud-computing provider. Figure 3 depicts the architecture and interaction among the main servers. 
Each one of these servers handles specific types of data and requests. The Production Server (PS) is an 
Nginx web-server, which processes web-client requests via a Ruby on Rails application. Account 
information such as user names, passwords, and account expiration dates, as well as curriculum 
taxonomy and state science standards are stored in the Production Database Server (PDBS). 
Anonymized events data generated by users interacting with the curriculum are stored in the Analytics 
Storage Server. These events are objects created by the Rails application on the Production Server (PS) 
and transmitted as encrypted packages via an HTTPS (Hyper Text Transfer Protocol Secure) session. At 
the end of the day, an automatic process harvests all events generated the previous day (stored in the 
Analytics Storage Server) and loads them into the Analytics Processing Server (APS). Once the data are 
loaded, a series of scripts are triggered, which calculate our analytics variables and aggregate results by 
user, school, and district levels. These results include information about visits by science standard, 
5E+I/A step, and curriculum element used. Similarly, the analytics variables are calculated at the end of 
each week, month, and year. Upon completion, the calculated data are transferred to the Dashboard 
Data Server (DDS). 

Although the current infrastructure is robust enough for our immediate needs, as we introduce more 
sophisticated analyses, such as machine learning, data mining algorithms, simulations, and 
computational linguistics, we envision the use of new methodologies in the realm of big data (Jagadish, 
2012; Pavlo, Paulson, & Rasin, 2009). We recently began experimenting with Hadoop (Venner, 2009) and 
the Map Reduce paradigm, an open source framework that enables distributed high-intensive 
computing for processing extremely large amounts of data. For instance, a process employing optimized 
queries and accessing pre-processed aggregated data for calculating twenty-five analytics variables for 
all teachers during the 2012-2013 school year takes approximately eight days to complete. Thus, we 
need to plan a scalable infrastructure as we begin calculating content- and user-based models derived 
from data spanning several years, across multiple schools, districts, and states along with recent user¬ 
generated data. Among the practical benefits of this big data approach to our K-12 curriculum are to 
allow the use of large clustering and machine learning algorithms for supporting personalized learning in 
the future. 
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Figure 3. Curriculum architecture and processes for storing, processing, and transferring data 

4. LEARNING ANALYTICS TECHNIQUES FOR MEASURING CURRICULUM USE 

The first research question that we posed is how researchers can utilize LA data to understand how and 
how much teachers and students use the curriculum. This information is useful both to the curriculum 
developers as well as to researchers investigating the curriculum's effectiveness. In this section, we 
focus on the first. The 5E+I/A inquiry model described above guides the analysis of all data collected 
because without this pedagogical grounding, the data patterns at best have no meaning, and at worst 
can be attributed the wrong meaning (Atkinson & Wiley, 2011; Long & Siemens, 2011). The model's 
underlying theory of learning, constructivism, is also used to answer important questions about 
curriculum implementation and how students learn best (Bybee et al., 2006). For example, 
constructivism and the 5E model assume that students learn best by exploring concepts before they are 
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presented with formal explanations of the content. An important task, then, is to examine whether 
teachers and students appear to use the steps and corresponding activities in this intended order. 


STEMscopes, however, also was created with the understanding that teachers want flexibility in how 
they teach. In practice, this means that a teacher can access and use any part of the lesson cycle when 
they deem most appropriate for their students, and most teachers do adapt the lesson cycle and the 
activities contained within them to their classroom needs (e.g., using components in a different order, 
changing them, or skipping some of them; Cho & Wayman, 2014; Hew & Brush, 2007; Keengwe, 
Onchwari, & Wachira, 2008; Koehler & Mishra, 2005; Lawless & Pellegrino, 2007; Leonardi, 2009; 
Orlikowski, 2000). This recognition creates a fundamental challenge for the curriculum designers who 
also believe that the structure of the 5E+I/A lesson cycle is the best learning path. 


This paradox creates a need to understand precisely what teachers and students are doing in their 
classrooms, and what these patterns of use mean for curriculum development and improvement. We 
have developed a strategy to complement a more traditional approach to measuring use or 
implementation with LA data (primarily analyzing teacher use). In the rest of the section, we describe 
two LA measures that examine the frequency and the ways in which teachers access the curriculum, and 
three ways we normalize the data to compare across teachers, schools, etc., and to conduct other 
analyses. 


All data were generated at the teacher level and come from a medium-sized school district in Texas 
during the 2012-2013 school year. 2 We selected this district because of an ongoing evaluation 
agreement and because it is one of the top districts in terms of curriculum usage. No identifiable 
information about district, school, or teacher was used. The original sample included 1,026 users and 53 
schools. We discarded 18 users since seven were categorized as administrators and 11 did not have any 
visits to the actual content. We also removed 6 schools (5 high schools and one alternative education) 
because they were only piloting the curriculum. The final sample included 36 elementary and 11 middle 
schools with a total of 898 and 88 teachers, respectively. 

4.1 A Pedagogy-Focused Approach to Measuring Use 

It is not enough to know how often the curriculum was visited. Because the curriculum was designed 
using a specific pedagogical model, the 5E+I/A lesson model, our analyses must try to uncover to what 
extent teacher use of the curriculum follows the model. Though there are multiple ways to measure 
how teachers use the curriculum with LA data, here we discuss two: the proportion of visits to each of 
the 5E+I/A steps and the proportion of visits that are "inquiry-focused." 

First, for each teacher the total number of visits to individual steps in the 5E+I/A framework was 
calculated. Teachers were then grouped according to their most visited step. The chart on the left in 
Figure 4 shows the percentage of teachers for whom each step was the most used. For instance, Explore 
was the most visited step for 360 (40%) elementary teachers and 13 (14%) middle school teachers; 
Engage was the most visited step for 242 (26%) elementary teachers, etc. Conversely, Acceleration was 


2 We have found that student use of the curriculum is relatively low, which we assume is due to the limited 
number of activities that students can complete online, as well as limited school access to computers and other 
digital devices. 
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the most visited for only 3 percent of elementary teachers and about 1 percent of middle school 
teachers. The chart on the right depicts the percentage of visits for each step. Explore was the most 
visited step for elementary schools teachers with 25 percent of the visits, followed by Engage with 22 
percent. For middle schools, the most visited step was Explain (21%) followed by Explore and Elaborate 
(18% each). What these charts mean is that, overall, most visits by elementary teachers are to the 
hands-on activities of the curriculum. In middle school, the same is true — the most visited step is 
Explore. Interestingly, middle school teachers access the intervention activities substantially more than 
do elementary teachers. 


Intervention 

Acceleration 

Evaluate 

Elaborate 

Explain 

Explore 

Engage 


Teachers' Most Visited Step 



5E + I/A Percentages of 
Total Visits 



■ M.S. visits 
N = 17,527 

■ E.S. visits 

N = 199,583 


Elaborate 


Explain 


Explore 


Engage 



0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 


0% 5% 10% 15% 20% 25% 30% 


Figure 4: Percentage of teachers for each one of the most used steps (left). 
Percentage of visits to each 5E+I/A step (right) 


Examining teacher use of the curriculum is quite complex and has continually required us to refine what 
we want to know, usually leading to more nuanced questions. Even the most basic question about 
frequency of use has many possible answers that require different approaches for calculating variables. 
For instance, we can capture overall use for each teacher who has an account, but this tells us nothing 
about how frequently they use specific components; it might simply be that they sign in and out. 
Therefore, we also calculated how frequently teachers use individual units (there is one unit for each 
state standard), different steps of the model, and other components within the curriculum (e.g., teacher 
background material). We also have drilled down further by weighting these statistics by the number of 
grades a teacher teaches or the number of science topics covered. In this way, we have begun to deal 
with questions of data normalization, which we will discuss more below. 

Questions of time introduce additional complexity. The analytics can tell us how long a teacher is logged 
in, when the first and last logins were, how long a teacher spends on each page, and on how many 
different days a teacher actually accesses the website. These can help normalize data as well as give us 
additional information about how teachers use the curriculum. For example, teacher activity for a 
sample of teachers in the district in our study throughout the 2012-2013 school year showed a great 
deal of variation in actual use. These data are summarized in Table 1 below. For elementary schools (36 
schools and 898 teachers), on average, teachers visited the curriculum 329 times, with a minimum of 2 
and maximum of 4,784 visits; the number of days on which teachers actually log in to access curriculum 
content averages 30 (minimum=l and maximum=175 days). 
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We also found variation in the time span during which teachers actively use their curriculum accounts 
(from first login to last login) for the period of interest. The range was from one day to 277 days, with an 
average of 181 days. We calculated the percent of the total time (in days) that teachers were actively 
logged into the website; this statistic ranged from 0% to 100% of the time with an average of 22%. We 
also adjusted this figure to take into account the number of grade levels the teachers teach; some teach 
only one grade, while others teach science to multiple grades. Finally, on average teachers visited a 
science standard 18 times, while the average of visits by grade level was 199 times. Results for middle 
schools are shown in the bottom part of Table 1. 


Table 1. Teacher data on visits to STEMscopes 



Total 

visits 

Number of 

different 

Time span 

Total time span 
weighted by 
number of 
unique days 
teacher logged in 
(%) 

Number of 

science 

topics 

taught 

Number of 

Avg. visits 

Avg. visits 


days 
teachers 
logged in 

for visits (in 
days) 

grade levels 
taught 

by science 
topic 

by grade 
level 

Elementary School 

Teachers N = 898 

Mean 

329 

30 

181 

0.22 

15 

2 

18 

199 

Median 

191 

21 

212 

0.14 

13 

1 

15 

128 

Minimum 

2 

1 

1 

0.01 

1 

1 

2 

2 

Maximum 

4,784 

175 

277 

1.00 

161 

9 

102 

1,672 

Middle School Teachers 

N = 88 

Mean 

302 

22 

161 

0.26 

16 

2 

15 

166 

Median 

122 

11 

194 

0.13 

13 

2 

11 

53 

Minimum 

3 

1 

1 

0.01 

1 

1 

2 

2 

Maximum 

3,897 

159 

277 

1.00 

90 

9 

169 

3,897 


The user analytics variables ranged substantially across schools. Among the main reasons for variation in 
the number of visits, we identified four: number of standards taught, number of grades taught, longevity 
of the accounts, and number of different days visited. Longevity represents the number of days elapsed 
since the user account was created. Therefore, older accounts would tend to have more visits, while 
newer accounts fewer visits. In the following sections we discuss the effect of these variables on the 
number of visits to the curriculum. 


IIC _ H( En+Ex + e p) 

Formula 1 

There are many ways to measure use from a pedagogical perspective, and we have developed various 
indices that allow us to conduct richer analyses of teacher use of the curriculum. A comprehensive 
description and use of these indices, however, is beyond the scope of this paper. The Inquiry Instruction 
Contribution Index (11C Index, depicted in Formula 1 below) is one way to measure use. To illustrate our 
approach, here we describe the application of the IIC to the total number of visits to the curriculum: 
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Specifically, the 11C measures the proportional contribution of the three most "inquiry-based" 5E steps 
of the lesson cycle (Engage, Explore, and Explain) to the overall curriculum usage (V) for each science 
standard. This index is important for pedagogical reasons: constructivism and the 5E model favour 
learning through hands-on, student-centred exploration, and we therefore have an interest in knowing 
what the proportion of teacher visits are to the more inquiry-based steps and activities. The index 
provides a summary statistic describing the level of inquiry in the classroom: a figure closer to 0 would 
suggest relatively little inquiry, whereas a figure closer to 1 would suggest more inquiry focus in a 
classroom. 


Curriculum Use and Inquiry Instruction Contribution 
(N=47 schools, 986 teachers) 

Elementary Middle School 



0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 

Inquiry Instruction Contribution (IIC) Index 


Figure 5: Boxplots depicting teacher visits to the curriculum (Y axis in log 10 scale) and Inquiry 

Instruction Contribution Index (X axis). 


Although it is difficult to determine a "perfect" use dosage, our curriculum design team suggests that 
between two-thirds and three-quarters of the total curriculum visits should be dedicated to the Engage, 
Explore, and Explain steps. In Figure 5, we compare curriculum visits (Y-axis) with the IIC index (X-axis) 
for each teacher across elementary (N=36 schools, 898 teachers) and middle schools (N=ll schools, 88 
teachers). The comparison indicates that the level of the inquiry focus varies according to school level. 


Table 2 shows the distribution of total visits and IIC for teachers in both elementary and middle schools. 
Elementary schools in this district have the highest level of total use with an average of 329 and 
maximum of 4,784, and most of that use clusters in the higher range of the IIC index, which suggests 
both that the elementary teachers visit the Engage, Explore, and Explain steps a fair amount, and that 
there might be a relationship between higher use of the curriculum and a more inquiry-focused 
classroom. In contrast, the middle school teachers have lower use with an average of 302 and maximum 
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of 3,897, and their IICs are more evenly distributed, that is, the frequency of teacher visits does not 
appear to be related to the inquiry focus of a teacher's classroom. 

Table 2. Distribution of total visits and IIC 


Elementary School 

Middle School 

Teachers 

Teachers 

Total 

Total 

visits IIC 

Visits IIC 


Minimum 

2 

0.00 

3 

0.00 

1st quartile 

73 

0.58 

41 

0.42 

Median 

191 

0.71 

122 

0.60 

Mean 

329 

0.69 

302 

0.58 

3rd quartile 

416 

0.83 

319 

0.70 

Maximum 

4784 

1.00 

3897 

1.00 


Broken down this way, the IIC data start to paint a picture of how teachers use the curriculum. These 
data might then be used to revise the middle and high school portions of the curriculum to see where 
improvements might be made to the more inquiry-based steps and activities. On the other hand, these 
data might be used to identify needs for professional development and coaching that can be embedded 
into the curriculum (e.g., middle and high school teachers may need support in figuring out how to 
integrate more hands-on activities into their lessons). 

4.2 Number of science standards taught 

In addition to creating LA-based measures of curriculum use, we must also ensure that the data are 
comparable across different units and levels of analysis. One important consideration is the number of 
science standards taught. Within their accounts, teachers can access all state standards throughout the 
school year. Because different districts and even schools may have their own curriculum scope and 
sequence, it is necessary to examine overall teacher use while taking into consideration the number of 
standards that have been taught. Without this form of normalization, a teacher in a district with one 
scope and sequence may appear to use the curriculum more than a teacher working under a different 
scope and sequence. 

Figure 6 illustrates this relationship. The total number of teacher visits is on the Y-axis (in log 10 scale) 
and the total number of science standards taught is on the X-axis. Each point in the graph corresponds 
to a teacher and the dotted line is the mean number of visits. The graph points to a weak but positive 
relationship: as the number of science standards accessed increases, so does the number of the overall 
visits. Without taking this relationship into consideration, we might have a biased understanding of 
curriculum use. 

4.3 Grade levels accessed 

A second variable that influences the total number of teacher visits to the curriculum is the number of 
grade levels that a teacher accesses. We have found in our research that teachers access standards from 
more than one grade level for different reasons: some teach science to more than one grade level, while 
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others pull activities from previous grade levels as a review, and still other teachers share accounts. 
Unfortunately, it is only possible to know why teachers access different grade levels if you ask them 
directly through a survey, interview, or focus group. It is possible, however, to control for the number of 
grade levels accessed as another way to normalize overall use: without this control, a teacher account 
that accesses numerous grade levels might have more visits to the curriculum, even though the teacher 
may not teach more science. Figure 7 depicts the weak but positive relationship between total visits (Y- 
axis, in log 10 scale) and number of grades taught (X-axis). Each point represents one teacher. 


Curriculum Visits and Number of Science Topics Taught 
Disaggregated by School Level 
(N=47 schools, 986 teachers) 



Figure 6. Total number of visits to the curriculum and number of science standards taught. 


Curriculum Visits and Number of 
Grade Levels Taught 
(N=47 schools. 986 teachers) 



Grade levels taught 


Figure 7. Total number of visits to the curriculum and number of grade levels taught. 
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A third consideration for data normalization is the longevity of teacher accounts. Longevity refers to the 
number of days elapsed between the creation of an account and a second given date (e.g., the end of 
the school year, the last day of use, etc.). Teachers do not always activate their online accounts at the 
same time, which we hypothesized would affect the total number of visits to the curriculum: teachers 
with "older" accounts should have a higher number of overall visits, but this might not mean that they 
teach more science. Figure 8 shows number of visits on the Y-axis in log 10 scale and longevity of the 
account in days on the X-axis as of the first day of the school year, disaggregated by school level. Values 
closer to the left are newer accounts (negative values indicate days elapsed since the beginning of the 
school year and the date when the account was created), whereas values to the right correspond to 
older accounts (positive values indicate the number of days that accounts have been active). Most of the 
schools show two clusters of account longevity, one around 0 days and the other between 550 and 600 
days. These clusters represent the creation of new accounts at the beginning of each school year. The 
plots do not show defined trend lines and therefore do not suggest a correlation between use and 
longevity. However, elementary schools created more new accounts once the school year had started 
than middle schools did. 


Curriculum Visits and Account Longevity 
(N=47 schools, 986 teachers) 

Elementary Middle School 



6 200 400 -200 6 200 

Account longevity 

Figure 8. Total visits to the curriculum and account longevity. 


4.5 Proportion of days visited 

Finally, another way to think about teacher use of the curriculum is to measure the time during which 
they have accessed it actively. To do this, we created a variable that represents the number of unique 
days a teacher has accessed the curriculum divided by the teacher's total log-in time span in days 
(calculated as the difference, in days, between the first and last day a teacher logged in). Values closer 
to 1.0 indicate a teacher who has logged in virtually every day; a value of 0.5 indicates a teacher who has 
logged in nearly half of the time they had an active account; and values closer to 0.0 indicate a teacher 
who seldom logged in. For example, a teacher who uses the curriculum on a daily basis will have more 
unique days than one who accesses it once a week. It is important to note that this measure does not 
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take into account the amount of content accessed within a single visit. In other words, it is possible that 
one teacher with more unique visits than another has accessed less content during each visit such that 
the two teachers have accessed the same amount of content, or such that the teacher with fewer 
unique visits actually has accessed more content. 


Figure 9 depicts the relationship between the total visits and the proportion of days visited. Total visits 
on the Y-axis are in log 10 scale, and the proportion of days visited on the X-axis ranges from 0.0 to 1.0. 
The elementary school graph shows two clusters, the first of which depicts a growing trend in total visits 
as the proportion of days visited increases (between 0 and 0.5). The second cluster shows a small 
number of teachers with 100% proportion but very low total visits (approximately less than 75 visits for 
the entire school year). Middle schools show a similar pattern, with the exception that the proportion of 
visits ranged approximately between 0.0 and 0.35. 


Total Visits Compared to Proportion of Days Visited 
(N=47 schools. 986 teachers) 



Proportion of days visited 


Figure 9: Total visits to the curriculum and proportion of days visited 


This kind of data helps curriculum designers understand if and how often the curriculum is being used; 
what content is most and least used; and the dosage of 5E+I/A steps they accessed. In future, we will 
also be able to relate these data to student learning outcomes, both those measured by assessments 
embedded in the curriculum (which students can take online) and by district and state standardized 
assessments. For school and district administrators, this information can shed light on understanding 
teaching practices and design professional development to identify best teaching practices and address 
deficiencies in instruction. 

5. LEARNING ANALYTICS AND DATA VISUALIZATION TO SUPPORT TEACHERS 
AND ADMINISTRATORS 

In addition to using the LA data to help curriculum developers, we also process the LA data in ways to 
support the end users of the curriculum — teachers and administrators, primarily. In our research on 
curriculum use, we have learned that use varies depending on the role of the end user. For example, 
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teachers were more interested in their own use and their students' progress, while campus-level leaders 
were interested in trends across classes, and district-level administrators wanted to know about use and 
progress across campuses (Snodgrass Rangel, Monroy, Bell, & Whitaker, 2013). Research also tells us, 
however, that educators often struggle to turn data into actionable information (Heer & Shneiderman, 
2012; Ingram, Seashore Louis, & Schroeder, 2004; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Lachat 
& Smith, 2005; Wayman, 2005; Wayman et al., 2010). 

Data visualization is an important tool that, in concert with strong theories, can help make sense of data 
(Duval, 2011). It is key, however, that the design of these visual interfaces be driven by teacher and 
administrator feedback. For example, several teachers with whom we have worked indicated that 
knowing what parts of the curriculum they had not yet used would help them know what they still 
needed to teach. In response, we created a heat map where colour intensity depicts the most and least 
used sections, with the goal of driving teacher attention to those sections. We have developed other 
data visualization technique, but covering all of them is beyond the scope of this paper. Here we 
describe three of them: timelines, heat maps, and a mastery tracker. 

5.1 Timelines 

Time-based visualizations can show when materials are accessed and for how long, as well as shifts in 
what science standards teachers access during a particular period of time. In this vein, timelines can be 
used to identify what teachers are teaching, when, and for how long. For example, one could see 
whether the life science standards are accessed before the physical science standards. The timelines can 
also shed light on the period of time, or duration, during which a teacher accesses a particular standard 
regularly. Pacing suggests the time elapsed between the completion of one standard and the beginning 
of a new one, which can help to identify standards that teachers may be overlapping or teaching at the 
same time (i.e., reviewing a set of standards while also introducing new standards). In the following 
paragraphs, we discuss two examples of the use of timelines. 

The duration of instruction for the science topic entitled "3.5AB- Classifying Matter" for a sample of 
teachers in one school is depicted in Figure 10. Each horizontal bar corresponds to one teacher and bars 
are sorted by the first day they were used. The position on the X-axis represents the first time a teacher 
accessed that science topic, the length of the bar indicates the duration in days. The graph shows that 
"Classifying Matter" was introduced early in the school year and most of the teachers accessed it for 
about three to four weeks. There are also a few long horizontal bars (spanning almost to the end of the 
school year), this could be an indication of teachers reviewing certain materials after those science 
topics had been originally covered. 

Another way to examine timelines is to explore how a teacher covered science topics throughout a given 
period of time. Figure 11 shows all science topics taught by one teacher. The Y-axis corresponds to 
science topics and the X-axis to time in days. This teacher covered a total of 23 science topics, of which 
21 belong to 3 rd grade and two to 2 nd grade, respectively. This proportion most likely suggests a 3 rd grade 
teacher accessing content from a lower grade for reinforcement purposes. Bar lengths can be clustered 
in three groups. The longest bars (five science topics), show duration of nearly the whole school year. A 
second cluster (three bars), shows duration between three and four months. A third cluster (five bars), 
shows duration between three and six weeks. The final cluster (nine bars) shows durations of 
approximately one week or less. 
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Duration of Science Topic Instruction 
by Teachers (in days) 

"3.5AB- Classifying Matter” 


m 

L, 



02-23-2012 04-13-2012 06-02-2012 07-22-2012 09-10-2012 10-30-2012 12-19-2012 02-07-2013 03-29-2013 05-18-2013 07-07-2013 

School Year 


Figure 10. Timeline depicting duration of instruction for all teachers teaching the science topic entitled 

"3.5AB- Classifying Matter" 


Instruction Time (in days) for a Teacher 
Throughout the School Year 

3.10C- Life Cycles 
3.10B- Inherited Traits and Learned Behaviors 
3.10A- Adaptations 
3.9B- Food Chains 
3.9AC- Environments 
3.70- Resources 
3.7B- Earth’s Forces 
3.7C- Earth's Landforms 
3.7A- Formation of Soil 
3.8CD- Space 
3.8B- The Sun and Water Cycle 
3.8A- Weather 
3.5C- Changes from Heat 
3.4A- Tools 
3.3D- Science Careers 
3.SAB- Classifying Matter 
3.3C- Models 
3.3AB- Scientific Reasoning 
3.2ABCDEF- Experimental Design 
3.IB- Conservation of Resources 
3.1A, 3.4B- Safety 
2.80- Objects in the Sky 
2.SA- Classifying Matter 

09-11-2012 10-31-2012 12-20-2012 02-08-2013 03-30-2013 05-19-2013 07-08-2013 

Figure 11: Timeline depicting duration of instruction of all science standards for one teacher. 


5.2 Heat maps 

Heat maps allow administrators and teachers to visualize how frequently different parts of the 
curriculum, including standards, steps, and activities, are accessed. Heat maps use colour or gray scale 
intensity to depict degrees of relationship between variables without focusing on exact values. In this 
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paper we discuss two cases of heat map use for our analysis. The first example (currently available to 
teachers on the STEMscopes dashboard) helps to understand curriculum usage at the science topic level 
across the 5E+I/A steps. In the second example, heat maps are intended for school or district 
administrators and offer overviews of instruction dosage for the 5E+I/A steps for individual teachers at 
the school level. Figure 12 depicts a partial view of curriculum use for 3 rd and 4 th grades in one school 
district (grade level is indicated by the numeric prefix in the TEKS column). Each row corresponds to one 
science topic. Blank cells indicate resources not available in STEMscopes. This graph shows high use for 
most topics in 3 rd grade and from topics 4.5A to 4.6A, while low use can be seen from topics 3.8D to 
4.4A. The most used steps are Engage and Explore, while Acceleration is the least used. The most 
frequently used standards are "3.8CD- Space" and "4.6A- Forms of Energy." Conversely, "4.3C- Models" 
is the least accessed. This type of visualization offers a quick overview of what has been taught and to 
what degree. 


•- Less 


TEKS Scop# 

3.68. 5.60 BeSTAARt Fore** and Motion 
3 6C Magnetom and Gravity 

3.7A Formation of So* 

3.7B Earvrt Forcaa 

3.7C Eartfva L a ndforma 

3.70 Resources 

3. «A Weetner 

3»8 The Sun *no Water Cycle 

3.6C0 Space 

3 80 ReSTAARt Planets 

3.9A RaSTAAR! Envronment* 

3.9AC Envronmentt 

3.96 Food Cnam 

4.10A Adapt*5on* 

4.106 mnemed Tran* ana learned B< 

4. IOC Ufa Cyoaa 

4.1 A. 4.46 Safety 

4.16 Conservation of Resources 

4 2ABC0EF Experimental De*»gn 

4 3AB Scant Ac Reasonng 


Total Overview Etaantlala Engage Explore Explain Elaborate Evaluate IntarvanOon 



ajx 

Moods 

| | 

22 


* 

7 

• 


• 


*-30 

Soence Career? 

[~2*T] 

*1 

IS 

37 

*6 

44 


>4 


4 4A 

T00O 


1S4 

M 

2* 

4* 

*2 


M 


4.5A 

Ctassfyrg Matter 


UN 

447 

72* 

1*2 

9*4 

496 

42* 

442 

4 58 

OangM from Meet 

r^i 

•54 

Tee 

4*0 

27* 

37* 

271 

23* 

It* 

4.50 

Mature* and Soutane 

i i 

IN 

24* 

44* 

2*2 

2*2 

2M 

214 

220 

4.6A 

Forms of Energy 

s 

IW 

it« 

1*64 

•» 

10*0 

704 

79* 

•1* 


*t 

91 

71 


17* 


Figure 12: Heat map depicting a partial list of STEMscopes content use for one district. Science 
topics included are for 3 rd and 4 th grades (indicated by the numeric prefix on the TEKS column). 

We then investigated whether teacher visits were evenly distributed across steps or concentrated on a 
specific step. Figure 13 depicts six gray-scaled heat maps for elementary schools. Each row represents 
one teacher and columns from left to right show total visits, percentage of visits for the most used step, 
and seven grayed-scaled columns indicating the proportion of visits for each of the 5E+I/A steps for that 
teacher. The heat map scale ranges from 0% (lighter gray) to 100% (black). Rows are sorted by total 
visits by teacher; hence, higher users are at the top, while lower users are at the bottom. 

Approximately three-quarters of teachers in "School El" have the Explore step as the most visited one, 
between 27% and 49%. As the total of number of visits decreases, the most visited step varies between 
Elaborates and Engages. With the exceptions of teachers U_365 and U_933, access to the Acceleration 
and Intervention steps is extremely low. Usage in "School E2" also depicts high activity in the Explore 
category; however, high activity is also seen in the Elaborate step, ranging from 31% to 70% for most of 
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the higher users. Usage in "School E3" indicates a more balanced proportion for the Engages and 
Explores sections for approximately the top nine high users; it also suggests some degree of use for the 
Evaluates section. In contrast to the other two schools, "School E3" shows some degree of activity for 
the Intervention step. 
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Figure 13. Heat maps depicting percentage of visits to each 5E+I/A step for teachers in three 

elementary schools. 


Heat maps with a high concentration of cells in one or two columns would be an indication of highly 
skewed distributions. Therefore, the majority of heat maps show that most teachers tend to access 
different steps of the curriculum in a balanced way. Especially encouraging is to observe that overall the 
Engage, Explore, and Explain steps are evenly accessed for the highest users in the three schools. 
Conversely, teachers with a high concentration on one step are those located at the bottom of the 
graphs. Although, this is not the intended use of the curriculum, these teachers have an overall low 
number of visits. The low use for the Acceleration and Intervention steps can be attributed to teachers' 
lack of training with differentiating instruction among their students or no timely actionable information 
in the current dashboard, which makes it more difficult for them to identify diverse intervention levels 
for students. 

The dosage of curriculum use for middle-school teachers shows sparser distribution and fewer teachers 
(see Figure 14). For instance, the highest user in "School Ml," uses Evaluates the most, with 30% of 
usage. Darker colours on the last column indicate teachers using Intervention elements (almost half of 
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teachers). Similar distribution can be seen in "School M2." Use patterns for "School M3" depict a 
concentration on Engage and Explore content for the top four highest users and a shift to Explain, 
Elaborate, and Evaluate content for the following two teachers. One exception is teacher U_789, who 
accessed Acceleration elements all the time. 
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Figure 14. Heat maps depicting percentage of visits to each 5E+I/A step for teachers in three middle 


schools. 


5.3 Mastery trackers 

Finally, we have created a way for stakeholders to visualize student progress towards mastery of 
learning standards. This information is crucial for several reasons, not least of which is that in the U.S., 
public school students (and teachers and schools) are held accountable for their learning at the end of 
every school year in various subjects on mandatory state tests. The mastery tracker shown in Figure 16 
below depicts whether individual students are making progress toward mastery on the required 
standards. 3 


The focus of the visualization is on progress because the mastery approach to student learning assumes 
that students should have more than one opportunity (or in many cases, as many opportunities as they 
need) to master a concept. Figure 16 shows a prototype of a dashboard with information for a class of 
4 th grade students. The columns show standards grouped by area (e.g., Organism and Environments), 
and the upward green arrows represent an improvement in performance from the previous assessment, 
while the downward red arrows indicate a decline in performance from one assessment to the next. 
Students are clustered by degree of intervention or acceleration they require, represented by the 
coloured cells. Students in the intervention group (yellow-coloured cells) are the ones who, although 
they are making progress, need a moderate amount of support from the teacher. Those in the intensive 
intervention group (the pink-coloured cells) require more help because they are not making sufficient 
progress towards mastery. Conversely, students in the acceleration group (green-coloured cells) can be 
challenged with more advanced activities from the acceleration step. This dashboard can help teachers 
better understand individual student needs, making it easier for teachers to differentiate and even 
personalize their instruction. 


3 The definition of mastery can be set by teachers, but usually the benchmark is 80% or higher on the final 
standards-based assessment. 
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4th Grade STAAR Progress Report 
Progress Monitor 
November 2, 2012 

Acceleration 

Intervetion 

Intensive Intervention 
Student Name 


Jack Hg. 
Taylor Gb. 
Ana B. 

Keara Da. 
Darren Gr. 
Dylan Ln. 
Ja3on Md. 
Caitlyn Kl. 
Alicia My. 
Mark Rd. 
Bailey Ml. 
Kennedy Bm. 
LaShell Yu. 
Morgan Su. 
Cole Gd. 
Jackson De. 
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Figure 16: Dashboard prototype depicting a partial list of 4 th -grade student progress for 
all scopes grouped by topic. Red arrows indicate grade drop from previous assessment, 
while green arrows show grade increase from previous assessment. 


6. HARNESSING LEARNING ANALYTICS TO MEASURE FIDELITY OF 
IMPLEMENTATION 

Research question 3 asks how LA data can be integrated into the evaluation of a curriculum. In this 
section, we discuss how we are integrating LA data into our curriculum evaluation work by using LA data 
as one measure of fidelity of implementation. In this section, we also detail the third prong of our LA 
data strategy; that is, how we pair the LA data with more traditional and qualitative data collection and 
analysis to create a holistic understanding of curriculum use and impact. 

In order to understand the impact of the curriculum on student learning, we relate the level and kind of 
curriculum use to different student outcomes (Duriak & Dupre, 2008; Song & Herman, 2010). It is 
essential to understand how stakeholders are using the curriculum in order to contextualize the impact. 
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For example, if there is little use of the curriculum, then it is not possible to attribute any change in 
student learning to use of the curriculum; the reverse also is true. The LA data are key here because they 
provide a detailed and multifaceted account of curriculum use, as we described above. We utilize the LA 
data as a complement to more traditional forms of measuring use, which we also describe below. And 
while it can be difficult to scale a high-quality qualitative data collection strategy, it is difficult, if not 
impossible, to really understand how an online curriculum, such as STEMscopes, is being used without 
spending time observing its use in classrooms and talking to teachers and students about their use. 

Typically, program evaluators collect data relating to the fidelity of implementation (or use) of an 
intervention in order to gauge the relative impact of the intervention on student learning. Dusenbury, 
Brannigan, Falco, and Flansen (2003) suggest that fidelity of implementation (FOI) should be measured 
in five ways: 1) adherence to the program; 2) dose or exposure to the program; 3) quality of program 
delivery; 4) participant responsiveness to the program; and 5) program differentiation (differentiation of 
components within the program). Other researchers have proposed different ways to conceptualize and 
measure FOI, but we focus on the previous five. In the rest of this section, we briefly describe how we 
incorporate LA specifically into evaluations, and then we discuss three other ways we collect data to 
measure impact. 

We use LA data to measure fidelity of implementation in four of the five ways described in the previous 
section. First, LA data provide a measure of adherence ; that is, whether a teacher or student is utilizing 
the curriculum in the intended way. LA data allow us to track precisely how a teacher has interacted 
with the program, including the proportion of inquiry instruction, as discussed in section four. Second, 
LA data can measure dosage, or how frequently and for how long a teacher engages with the 
curriculum. Third, LA data can be utilized to measure how engaged a teacher is with the curriculum by 
looking at, for example, time spent on a component or how quickly a teacher moves through 
components. Fourth, LA data can be used to measure program differentiation, or the impact of different 
components of the curriculum on student learning. This can be done by examining how a teacher 
interacts with various components, and weighing these data with other dosage data to estimate their 
impact on student learning outcomes, such as interim assessment scores as well as end-of-year test 
scores. 

6.1 Using teacher surveys to measure fidelity of implementation 

A common way to gauge teacher use of a curriculum (or other intervention) is to ask them to self-report 
in a survey. As part of our ongoing effort to evaluate the use and impact of the STEMscopes curriculum, 
in the fall of 2012 we created and administered a survey to 755 elementary school teachers in a large 
urban school district where some schools use the curriculum. The survey, containing 80 items, was 
created based on existing surveys about technology and data use. While some of the items are specific 
to the STEMscopes curriculum and its particular offerings, others are generalizable to other technology 
or curricula. The survey asks teachers about the following concepts: teacher notions of data and data 
use, how teachers use STEMscopes, attitudes toward STEMscopes, challenges to using STEMscopes, as 
well as district and school support for using STEMscopes. The survey was piloted and validated in the fall 
of 2012 (See Snodgrass Rangel, et al. 2013 for more information on the survey and for more detailed 
findings), 210 teachers responded to the survey, a response rate of 28%. 
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As part of this study, and as an example of how to combine these two kinds of data, we related teacher 
survey responses to their LA data to see how closely they were related. Specifically, we used correlation 
analysis on the associations between these three variables and several user analytics variables. We also 
created and used four variables from the LA data we had access to at that time: 


1. The number of visits to pages containing instructional content for any of the seven steps, weighted 
by the number of grade levels accessed by the teacher's account (continuous variable) 

2. The number of visits to pages containing instructional content for any of the seven steps, weighted 
by the number of objectives the teacher had taught to that point (continuous variable) 

3. A categorical variable (created using quartiles) measuring the un-weighted number of visits to pages 
containing instructional content for any of the seven steps (l=low use, 2=medium low use, 
3=medium high use, 4=high use) 

4. A categorical variable (created using quartiles) measuring the number of visits to pages containing 
instructional content for any of the seven steps, weighted by the number of grade levels accessed by 
the teacher's account (l=low use, 2=medium low use, 3=medium high use, 4=high use). 

First, we correlated teacher responses to the survey question asking how often they utilized the 5E+I/A 
steps in the suggested order and an analytics variable that measured how frequently teachers actually 
used all seven steps in the suggested order. Second, we correlated teacher responses to how often they 
utilized the 5E+I/A steps in the suggested order and the frequency with which teachers used the 5E 
(without the I/A) steps in the suggested order. We found that while teachers reported using the 
curriculum for some to most of the standards they taught, the LA data told a different story (see Table 
3). 


Table 3. Descriptive Statistics for Reported and Actual Use of STEMscopes 


Mean SD Range 


Reported Use 

Average use of teacher components 

Average use of 5E+I/A steps 

Average use of 5E+I/A steps and components in 

order 

2.70 

2.48 

2.62 

0.76 

0.64 

0.60 

1.00-4.00 

1.00-4.00 

1.00-4.00 

Actual Use 




Number of 5E visits weighted by number of grades 

taueht 

103.42 

125.84 

1.00 - 1084.00 

Number of 5E visits weighted by number of 

objectives taught 

20.09 

13.36 

2.00 - 75.00 

Average number of visits to 5E+I/A steps 

(categorical) 

2.49 

1.13 

1.00-4.00 

Average visits to 5E+I/A steps, weighted by number 
of grades taught (categorical) 

2.24 

0.83 

1.00-4.00 
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In the correlation analyses, we found that while almost all of the correlations were positive, very few 
were statistically significant relationships. The two that were significantly (but weakly) related were 
between the categorical variable measuring the average number of visits to a page containing 
instructional content from any of the seven steps (l=low use, 4=high use) and the reported average use 
of teacher components (r(114)=0.195, p<0.05), and the reported combined average use of the teacher 
components, the 5E+I/A steps, and the frequency with which a teacher uses the steps in the appropriate 
sequence (r(114)=0.219, p<0.05). The results of the correlation analysis can be found in Table 4 below. 


Table 4. Results of correlation analysis for reported and actual use 


User analytics 


Reported average 
use of teacher 
components 

Reported 
average use of 
5E+I/A steps 

Average use of 
5E+I/A steps and 
components in 
order 

Number of 5E visits weighted by 

Pearson Correlation 

0.160 

0.030 

0.140 

number of grades taught (N=114) 

Sig. (2-tailed) 

0.090 

0.780 

0.130 

Number of 5E visits weighted by 

Pearson Correlation 

0.090 

-0.050 

0.060 

number of objectives taught (N=114) 

Sig. (2-tailed) 

0.330 

0.570 

0.560 

Average number of visits to 5E+I/A 

Pearson Correlation 

0.195 * 

0.140 

0.219 

steps (categorical) (N=114) 

Sig. (2-tailed) 

0.040 

0.140 

0.020 

Average visits to 5E+I/A steps, weighted 

Pearson Correlation 

0.060 

-0.040 

0.030 

by number of grades taught 

Sig. (2-tailed) 

0.550 

0.710 

0.730 


(categorical) (N=113) 

* Correlation is significant at the 0.05 level (2-tailed) 


We also correlated teacher survey responses about how frequently they use the 5E+I/A steps in the 
suggested order with the frequency with which they actually used all of the 5E+I/A steps in the 
suggested order (based on the data analytics). The results showed no significant association between a 
teacher's reported use and actual use of the 5E+I/A steps in the suggested order (r(lll) = -0.02, p = 
0.84). Second, we conducted a correlation between teacher survey responses about how frequently 
they use the 5E+I/A steps in the suggested order with the frequency that they actually used just the 
steps in the suggested order. The results showed no significant association between a teacher's 
reported use and actual use of the 5E steps in the suggested order (r(lll) = 0.02, p = 0.83). In other 
words, there appeared to be no relationship between perceived and actual use regarding the 5E+I/A 
steps in the order intended by the curriculum designers. 

The question that these two analyses raise, of course, is which measure of teacher use is correct? The 
answer we have settled on is that neither is "wrong" or "right," and that therefore they must be used 
jointly to piece together a balanced picture of teacher use. In other words, data from different sources 
must be triangulated. In the next two sections, we describe two more ways to collect data for the 
purpose of creating a more nuanced picture of curriculum use. 
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5.4 Using teacher focus groups and interviews to measure fidelity of 
implementation 


The research team also conducted teacher focus groups to understand how teachers use the curriculum. 
The benefit of conducting focus groups is that we are able to engage directly with teachers and ask them 
about their use of the curriculum. We can follow up on their answers and probe more deeply for 
explanations of certain behaviours. In this way, we can look at data based on access to the curriculum 
next to data on what the teachers do once they have accessed the curriculum and are in their 
classrooms teaching students. 


The focus groups we conducted were separate from the external evaluation discussed in previous 
sections. Instead, they have been part of internal research efforts. For example, in the fall of 2012, the 
research team asked the principals at two schools that use STEMscopes to put together a group of 3-5 
teachers to meet with us. A total of 11 teachers participated (see Table 5). Of these teachers, all but two 
were self-contained teachers who taught science in addition to other subject areas. Of the 11 teachers, 
three were novices (first year), one had medium experience (2-5 years), four were advanced (6-20 
years), and four were veterans (21+ years). 


Table 5. Description of focus group participants 




Grades 

Grades 

Novice 

Medium 

Advanced 

Veteran 

Total teachers 

Science only 

K-2 

3-5 

teacher 

experienced 

teacher 

teacher 

11 

2 

9 

2 

3 

1 

4 

4 


The team created an interview protocol to guide the focus group. The interviews were semi-structured, 
which allowed the conversation to develop organically. The protocol asked teachers how they felt about 
teaching science, using technology, using STEMscopes, and what challenges they faced when using 
STEMscopes to teach science. Here, we only share a few findings as they relate to use of the curriculum 
with the intent of demonstrating the type of findings that can be extracted from direct conversations 
with the curriculum's stakeholders. 

5.4.1 Uses of STEMscopes by teachers in grades K-2 

The majority of participating teachers were K-2, self-contained teachers. Though all of the teachers 
were very positive about science and reported enjoying teaching science, it was clear that they had very 
little time to teach science, and so their use of STEMscopes and other science resources was quite 
limited. According to the teachers, they were only able to teach science two to three times a week, and 
this relatively short time was split between hands-on experience, such as an experiment, and science 
vocabulary. In fact, we specifically asked the teachers how often they were able to do science; they 
reported that they were working on a hands-on experience with their students almost every week. 

Vocabulary, however, was the most frequently mentioned use of the curriculum; the kindergartners 
draw pictures of science concepts and the second graders make vocabulary cards with pictures, labels, 
and definitions. 4 STEMscopes' vocabulary materials were particularly helpful for the bilingual teachers, 


4 Most frequently mentioned does not necessarily mean the most frequently used. The analytics data suggest that 
K-2 teachers access the Explore step the most, indicating they do hands-on activities with their students. 
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most of whom teach science as an English as a Second Language (ESL) class where the focus is as much 
on learning English as it is on learning content. Also included in the discussion was the use of science to 
teach literacy. The teachers described spending most of their time on math and literacy to prepare 
students for the 3 rd grade state tests, and so science often became another way to work on literacy. A 
teacher from one school told us how her students read science stories, saying, "this week... there is a 
story about plants, so then you know they are identifying roots, petals, and stems, all that stuff." Other 
teachers described how the students kept science journals where they pasted in passages and diagrams 
to practice new vocabulary. 

5.4.2 Uses of STEMscopes by teachers in grades 3-5 

Teachers in the upper-elementary grades (particularly 5 th grade) had more time for science instruction, 
often more than an hour a day, which allowed them to utilize more parts of the curriculum. However, 
the teachers still expressed the need for more time to cover all of the material and review material from 
previous grades, where students often receive little science instruction. One of the teachers described 
how she strives to use all of the options contained within the curriculum, saying, "I wouldn't say [I use it] 
to its maximum but I would say as close as I can possibly get within the given timeframe that I am 
allotted." 

These two teachers indicated that they try to follow the curriculum closely by doing one "E" step on 
each day and using as many of the assessments as they have time for. One teacher confided that, "I 
don't use every experiment that you guys [have]...I use most of them though, but I try to make sure that 
I, we do an experiment [from STEMscopes] every Tuesday and then I try to do one on Wednesdays also, 

I try two days of complete experiments." In this way, the teacher generally is able to meet the student 
needs for hands-on learning and intervention at the end of the week for those students who did not 
demonstrate mastery on the post-assessment. Both of the teachers also told us that, for the most part, 
they complete each of the 5E steps in order, though one of the teachers told us that she begins each 
new unit with direct instruction and note-taking. Our observations of both classrooms confirmed that 
both teachers begin units with direct instruction and note-taking. 

Finally, the teachers rely primarily on STEMscopes for their actual lessons while following the district's 
scope and sequence, which directs them to teach the standards in a certain order. One of the teachers 
told us how she reconciles using a curriculum not officially adopted by the district: "I just read the 
teacher background and I just make sure that I am teaching in the sequence that [the district] is giving 
me to teach... I use the teacher background and all the activities from STEMscopes." 

5.5 Using classroom observations to measure fidelity of implementation 

Through classroom observations, we have seen how teachers and students interact — and do not 
interact — with the online curriculum. Observations, though time-consuming, are important because 
they allow for comparing and classifying interactions. Teachers often unwittingly over-report how often 
they use the curriculum, and observations can cut through the problems of self-reporting and 
perception. 

In that same internal evaluation, we observed two teachers at different schools as they taught science. 
Each of the teachers we observed taught science only and was a very experienced teacher. One of them 
taught science to grades 3-5, while the other only taught 5 th grade science. The teacher who taught 3 rd - 
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5 th grade science had 18 years of experience as a teacher and instructional coach (though not in 
science), and this was his/her first year teaching science only. Furthermore, his/her previous teaching 
experience was in l st grade. This teacher also helps to translate curriculum for STEMscopes into Spanish, 
making him/her very familiar with the curriculum. The other teacher had taught for 22 years, nine of 
them as a science teacher exclusively. This teacher also had previous experience teaching middle-school 
math. During the classroom observations, we took extensive field notes, which we later coded using an 
observation protocol created for science classrooms. Based on these notes, we created long and 
detailed "inquiry profiles" that described how teachers used the curriculum and level of inquiry in each 
teacher's classroom. Here, we report only general findings. 

We learned a great deal about the context and quality of implementation from the observations. For 
instance, we noted that in these classrooms, there were no more than two computers available to 
students, and the teachers either printed materials from the website, or projected the website onto a 
whiteboard. This finding can help explain variation in teacher use — some teachers may have more 
technology available to them than these teachers did. Similarly, we observed that while some teachers 
followed the curriculum very carefully, others used fewer of the activities and instead incorporated 
other available resources, and others still utilized the steps and activities out of the intended order. In 
short, the observations reveal "the face" behind the "clicks" that LA data show us. 

6. CONCLUSIONS AND FUTURE WORK 

LA has the potential to bring important changes to K-12 education research and development if 
implemented with a pedagogically sound, mixed methods strategy, such as the one described in this 
paper. Incorporating the LA data into our ongoing research on curriculum has enabled us to dig deeply 
into patterns of use and to ask nuanced questions that relate to the pedagogy underpinning the 
curriculum and its effectiveness. We have also learned that on their own, LA data paints an incomplete 
picture of what teachers do in their classrooms with the curriculum. By bringing in more traditional 
measures of use, such as surveys, interviews, and classroom observation, we were able to contextualize 
some of the findings gleaned from the LA data. 

As we have improved our technical capacity to work with the LA data, we have also created new ways to 
make the data accessible to the curriculum's key stakeholders, namely teachers and administrators. 
Data visualization has been the most powerful tool we have put into practice so far to translate data 
into information, and future work will examine the utility of these tools as well as develop new tools 
based on feedback. 

There also are several hurdles that must be overcome to ensure that these new data collection and 
analysis tools do, in fact, fulfill their promise. First, schools and districts must address the gap in access 
to technology. While some schools have achieved one-to-one computing, most schools are not even 
close to this goal, and this has profound implications for our ability to collect reliable analytics data. Our 
conversations with and observations of teachers revealed that teachers and students often share 
accounts, and that students are limited in the activities they can complete online. This, in turn, means 
that the analytics data we collect may not be reliable: for some teachers the usage analytics may 
accurately portray their use, while for others, they may not. 
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A second and equally important challenge is time. Teachers often do not have time to incorporate 
technology and online activities into their regular instruction. It is time-consuming to plan for this 
incorporation, especially because it often requires the teacher to learn new software or strategies, and 
because it requires the teacher to create new classroom and management routines. The result of this 
hurdle is that the computer systems that should be collecting analytics data are not fully implemented 
— or are not implemented at all, which degrades the quality of data we can collect. 

In our future work, we will include curriculum elements in each of the 5E+I/A steps and investigate the 
effects of the characteristics and properties of those elements in curriculum use. Similarly, we will 
disaggregate use by science standard, skills, and concepts covered on each curriculum step. These new 
scenarios will offer a deeper and richer picture of science teaching and learning in the K-12 realm. We 
will conduct studies to investigate sequences of visits, pacing, and patterns of use at different points in 
time. This in turn will help us to answer questions related to curriculum use changes throughout the 
school year and consistency of use across science standards, schools, and districts, with implications for 
targeted professional development for teachers and curriculum design. We will also examine specific 
use patterns and relate them to student learning outcomes. These, no doubt, will present the team with 
fresh challenges as some components of the curriculum may turn out to be more effective than others, 
and these results may confound the relationships we hypothesize to exist. 

Though not a panacea for the challenges facing K-12, a strong LA data strategy can help researchers and 
educators alike make sense of the large amounts of data generated by the increased use of mobile 
devices, computers, and other technology in classrooms. With an improved understanding of data, 
students can learn from their own mistakes. Teachers can improve their lessons, work with students 
who need extra help, and provide extensions for those ready to move on. Analysts can improve the 
curriculum and the resources available to teachers and students. The strategy described in this paper 
was developed to analyze detailed usage patterns of the online science curriculum, STEMscopes. This 
approach enables analysis of large amounts of data and opens the possibilities to complex undertakings 
such as the creation of personalized learning environments. 
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